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Errata  Sheet 


Page  i,  paragraph  1,  line  15:  change  word  "health  care"  to  "patient." 

Page  i,  paragraph  3,  change  "Group  Medical  Officers"  to  "General  Medical 
Officers . " 

Page  7,  paragraph  2,  line  16:  The  sentence  "The  Respondents  thought  the 
diagnosis  of  nonspecific  abdominal  pain  was  confusing  and  should 
therefore  be  deleted  from  the  system"  should  have  appeared  on  page  8, 
paragraph  5,  line,  2. 
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Summary 


Problem:  U.S.  Navy  Independent  Duty  Corpsmen  (IDCs)  aboard  small  ships 
and  submarines  are  responsible  for  the  diagnosis  and  treatment  of 
patients  while  at  sea.  Although  IDCs  safely  manage  the  majority  of  the 
patients,  some  life-threatening  illnesses  require  IDCs  to  recommend  that 
patients  be  evacuated  to  a  shore-based  facility  for  treatment.  At-sea 
evacuations  of  patients  are  dangerous  to  the  patient,  expensive,  and  may 
compromise  the  mission  of  the  ship  or  submarine.  An  IDC's  decision 
regarding  the  need  to  evacuate  a  patient  may  be  complicated  by  an 
inability  to  conduct  certain  tests  because  of  a  lack  of  shipboard 
laboratory  facilities.  Furthermore,  mission  requirements  frequently 
preclude  communicating  with  shore-based  physicians  for  consultation. 
Usually,  a  single  IDC  is  assigned  to  each  small  ship  or  submarine  and  is 
tasked  with  all  clinical  and  related  health-care  duties  for  embarked 
personnel.  A  system  is  needed  that  would  aid  IDCs  in  the  evaluation, 
treatment,  and  management  of  health  care  aboard  ships  and  submarines. 

Objective:  The  objective  of  this  study  was  to  gauge  the  practical 
implications  of  IDCs  employing  a  user-friendly  computer-assisted  medical 
diagnostic  (CAMD)  system  to  aid  in  the  diagnosis  of  disease  and  to 
perform  tasks  related  to  the  treatment,  evaluation,  and  management  of 
patients . 

Approach:  The  Navy  Computer-Assisted  Medical  Diagnosis  (NCAMD)  system 
was  installed  aboard  U.S.  Navy  ships  and  submarines  with  GMOs  or  IDCs 
attached.  Both  the  Group  Medical  Officers  (GMOs)  and  IDCs  were  trained 
in  the  functioning  and  use  of  the  NCAMD  system,  and  given  the  software 
to  use  and  evaluate  during  deployment ,  When  they  returned  from 
deployment,  the  GMOs  and  IDCs  were  surveyed  and  interviewed  to  determine 
their  opinions  of  the  NCAMD  system. 

Results :  The  results  show  that  NCAMD  software  was  user-friendly,  easy 
to  learn,  valuable  for  training,  and  could  significantly  contribute  to 
health-care  delivery.  However,  NCAMD  was  installed  on  the  ships' 
available  computer  hardware,  which  was  reported  as  slow,  and  inhibited 
the  continuous  use  of  NCAMD.  NCAMD  users  stated  that  the  addition  of  a 
variety  of  modules,  structured  data  sheets,  treatment  and  reference 
materials,  and  peripheral  devices  would  increase  value  and  usage  of  the 
system. 

Conclusions :  The  NCAMD  software  is  a  cost-efficient  and  user-friendly 
package  that  is  easy  to  learn  and  is  of  particular  value  for  training  in 
the  diagnosis  of  chest  and  abdominal  pain.  However,  the  delivery  of 
health  care  at  sea  would  significantly  improve  if  the  computer  hardware 
were  upgraded  to  current  industry  standards.  Also,  adding  various 
computer  peripheral  devices,  structured  forms,  and  reference  materials 
to  the  at-sea  clinician's  resources  could  enhance  shipboard  health  care. 


Introduction 


Numerous  computer-assisted  medical  diagnosis  (CAMD)  software  programs 
have  been  created  to  aid  clinicians  in  the  diagnosis  of  disease  (Waxman 
&  Worley,  1990).  These  programs  have  been  tested  extensively  and  their 
diagnostic  accuracy  has  been  found  to  be,  for  certain  diseases,  better 
than  that  of  the  average  clinician  (de  Dombal,  1989;  McAdam  et  al.,  1990; 
Patterson-Brown  &  Vipond,  1990) .  CAMD  programs  have  also  proven  to  be 
of  value  in  the  area  of  clinical  education  because  they  improve  data 
collection  procedures  by  imposing  a  structure  upon  the  gathering  of  signs 
and  symptoms  of  illness  (Lawrence,  Clifford,  &  Taylor,  1987).  Despite 
the  well -documented  advantages  of  CAMD  programs,  there  has  been  little 
use  of  the  programs  (de  Dombal,  Dallos,  &  McAdam,  1991),  Therefore,  it 
appears  that  CAMD  programs  are  not  used  for  reasons  other  than  their 
diagnostic  accuracy  or  their  value  to  clinical  education. 

CAMD  developers  in  the  early  1960s  believed  that  their  systems  would 
eventually  replace  clinicians.  Their  omnibus  systems  were  overly 
ambitious,  especially  when  the  unwieldy  hardware  and  the  available 
software,  limited  by  the  computers'  memory,  are  taJcen  into  consideration 
(Dallos  &  Cunningham,  1990).  Through  the  1980s,  hardware  capabilities 
lagged  behind  the  development  of  software,  which  made  the  use  of  CAMD 
systems  time-consuming  and  impractical.  However,  with  the  recent 
development  of  faster  central  processing  units  (CPUs) ,  the  practical  and 
efficient  use  of  CAMD  software  is  now  possible,  which  may  lead  to  greater 
acceptance  of  CAMD  systems.  Instead  of  CAMD  systems  replacing  clini¬ 
cians,  however,  it  has  been  found  that  CAMD  systems  that  consider  finite 
clinical  problems  can  augment  the  diagnostic  slcills  of  clinicians,  teach 
and  guide  inexperienced  clinicians,  and  aid  in  promoting  a  standardized 
method  for  evaluating  patients  (de  Dombal  et  al.,  1991;  Wellwood, 
Johannessen,  &  Spiegelhalter ,  1992).  Overall,  studies  have  shown  that 
CAMD  systems  can  complement  the  clinicians'  previously  acquired  skills 
(Dallos  &  Cunningham,  1990;  de  Dombal,  1990a;  Waxman  &  Worley,  1990). 

For  more  than  a  decade,  the  Naval  Submarine  Research  Laboratory 
(NSMRL)  developed  CAMD  modules  for  use  by  Navy  Independent  Duty  Corpsmen 
(IDCs)  and  Group  Medical  Officers  (GMOs) .  GMOs,  physicians  stationed 
aboard  large  Navy  ships,  are  responsible  for  the  health  care  of  all 
attached  personnel  and  the  supervision  of  all  IDCs  under  their  level  of 
command.  Quite  frequently,  lone  IDCs  assigned  to  Navy  vessels  are 
responsible  for  the  health  care  of  more  than  2  50  personnel  (Nice,  1984)  . 
IDCs  are  specially  trained,  nonphysician  health-care  providers  who  are 
responsible  for  the  diagnosis  and  treatment  of  patients  who  incur  trauma 
and  illness  while  at  sea  aboard  submarines  and  small  ships.  Although  the 
IDCs  safely  manage  the  majority  of  patients,  some  life-threatening 
illnesses  require  the  IDCs  to  recommend  that  the  patients  be  evacuated 
to  a  large  ship  or  shore-based  activity.  The  evacuation  of  patients 
while  at  sea  is  dangerous  to  the  patient,  expensive,  and  may  compromise 
the  mission  of  the  submarine  or  ship.  The  IDC's  decision  of  whether  to 
evacuate  may  be  complicated  by  a  lack  of  laboratory  facilities  and 
diagnostic  equipment  and  by  an  inability  to  communicate  with  physicians, 
imposed  by  mission  requirements.  Therefore,  CAMD  modules  were  developed 
to  assist  IDCs  with  the  diagnosis  and  treatment  of  patients  who  present 
with  abdominal  pain,  chest  pain,  trauma,  dental,  ophthalmologic,  or 
psychiatric  complaints  (Caras,  Southerland,  &  Fisherkeller ,  1989) .  These 
diagnostic  categories  have  been  shown  to  be  the  most  common  areas  wherein 
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IDCs  and  GMOs  have  required  assistance  (Nice,  1984)  .  The  performance  of 
NSMRL  CAMD  modules  for  chest  pain  and  for  abdominal  pain  have  been  tested 
at  sea  and  found  to  be  of  value  when  the  patient's  diagnosis  is  in  doubt 
(Luria,  Southerland,  &  Stetson,  1990;  Perrotta  &  Stetson,  1992)  . 

IDCs  are  also  tasked  with  a  variety  of  professional  and  military 
duties.  Their  professional  duties  include  cleanliness  and  environmental 
hazard  inspections,  physical  examinations,  health  education,  and  routine 
and  emergency  sick  call.  Additionally,  they  may  be  assigned  collateral 
duties  and  must  perform  routine  military  duties.  Their  professional 
duties  require  the  completion  of  periodic  hard  copy  reports  that  are  sent 
to  higher  echelons  of  command.  Consequently,  much  of  their  time  is 
occupied  with  report  writing  and  handling  of  forms. 

Recently,  the  Naval  Health  Research  Center  (NHRC)  integrated  the 
patient  functions  of  the  NSMRL  CAMD  module  shells  with  those  of  the 
Shipboard  Non-Tact ical  ADP  Program  (SNAP)  Automated  Medical  System  (SAMS) 
and  created  one  diagnostic  package.  Because  this  system  has  been 
designed  to  be  used  in  Navy  shipboard  environments,  it  is  referred  to 
as  the  Navy  Computer  Assisted  Medical  Diagnosis  (NCAMD)  system.  SAMS  has 
been  installed  at  more  than  500  Navy  facilities.  NHRC  programmed  the 
enhanced  patient  encounter  module,  incorporated  it  into  SAMS,  and  added 
the  following  capabilities:  (1)  computer-assisted  diagnostic  support,  (2) 
an  ICD-9-CM  diagnosis  look-up  table,  (3)  an  SF600  facsimile,  (4)  a 
medical  evacuation  data  entry  form,  (5)  an  expanded  patient  encounter 
form,  and  (6)  monthly  morbidity  reporting  functions.  NHRC's  overall  goal 
was  to  enhance  health-care  delivery  by  reducing  the  IDCs'  routine  work 
load  and  by  providing  adjunctive  clinical  assistance  in  the  form  of 
NCAMD.  To  this  end,  NHRC  created  a  flexible,  computerized,  and  user- 
friendly  management  information  system  that  would  integrate  a  structured 
means  of  sign-symptom  gathering,  diagnostic  assistance,  and  the 
production  of  standard  reports  and  forms.  This  system  will  allow  the  IDC 
to  more  efficiently  and  accurately  complete  many  tasks  that  had 
previously  been  done  manually,  thereby  allowing  the  IDC  to  devote  more 
time  to  direct  health-care  delivery  (Ly,  Merrill,  &  Pearsall,  1993). 

De  Dombal  (1990a)  outlined  the  following  difficulties  associated 
with  the  implementation  of  CAMD  programs:  (1)  clinicians  are  unwilling 
to  believe  that  CAMD  programs  can  enhance  their  performance,  although  the 
complexity  of  clinical  medicine  has  increased  dramatically  and  ample 
evidence  supports  the  claim  that  diagnostic  skills  can  be  improved  with 
CAMD  programs;  (2)  CAMD  programs  need  to  be  developed  that  prescribe  that 
structured  methods  be  used  to  gather  patient  data  and  case  histories;  (3) 
user-friendly  interfaces  need  to  be  developed  (de  Dombal  stated  that  the 
development  of  user-friendly  interfaces  in  the  field  of  clinical 
computing  requires  "urgent  action");  (4)  the  debate  concerning  which 
system  is  best,  probabilistic  or  deterministic,  needs  to  stop,  since  both 
systems  appear  to  produce  similar  results;  (5)  the  problem  of  which 
information  to  give  the  clinician:  a  diagnosis,  a  list  of  possible 
diseases  with  their  probabilities,  or  a  suggestion  of  what  the  diagnosis 
may  be  with  options  that  may  be  investigated;  (6)  the  problem  of 
convincing  clinicians  that  their  performance  improves  because  they  have 
a  disciplined  method  for  data  collection,  they  have  learned  common 
terminology,  the  computer  is  a  tutor  and  provides  feedback,  and  the  CAMD 
system  incorporates  practices  that  have  been  shown  to  be  effective  by 
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experienced  clinicians;  and  (7)  the  lack  of  a  computer-literate  work 
force,  educated  in  the  prudent  use  of  information  technology. 

This  study  was  done  to  gauge  the  practical  implications  of  employing 
a  user-friendly  CAMD  system  to  aid  in  the  diagnosis  of  disease  and  to 
perform  tasks  related  to  the  treatment  of  patients.  For  many  reasons  the 
intended  users  of  CAMD  systems  are  not  using  CAMD  programs.  Although  de 
Dombal  (1990a)  has  suggested  various  problems  associated  with  the 
implementation  of  CAMD  systems,  few  studies  have  been  done  that  have 
solicited  potential  users'  opinions. 

Method 

There  appears  to  be  ample  evidence  for  the  implementation  of  CAMD 
systems  aboard  ships  and  submarines.  For  example,  they  have  been  shown 
to  have  numerous  benefits  for  clinicians  and  patients  (Dallos  & 
Cunningham,  1990;  de  Dombal,  1990a;  Waxman  &  Worley,  1990),  their 
placement  and  use  aboard  U.S.  Navy  ships  and  submarines  suggest  that 
health-care  delivery  is  enhanced  (Nice,  1984),  and  they  have  been  found 
to  be  of  value  at  sea  when  the  patient's  diagnosis  is  in  doubt  (Luria  et 
al.,  1990;  Perrotta  &  Stetson,  1992).  To  evaluate  the  need  for  the 
implementation  of  NCAMD,  the  system  was  installed  aboard  20  U.S.  Navy 
vessels  (10  ships  and  10  submarines)  with  IDCs  or  GMOs  attached.  Some 
of  the  vessels  have  more  than  one  IDC  embarked.  All  of  the  NCAMD  systems 
were  installed  by  NHRC  personnel  or  by  extensively  trained  IDCs  or  GMOs. 

Apparatus 

Most  of  the  ships  and  submarines  on  which  NCAMD  has  been  installed 
have  computers  with  286  CPUs.  These  systems  are  comparatively  slow; 
therefore,  many  ships  and  submarines  are  in  the  process  of  upgrading 
their  systems  to  486  CPUs. 

The  user  interface  for  NCAMD  has  been  implemented  as  a  window-oriented 
environment.  The  use  of  NCAMD  has  been  detailed  in  Ly  et  al .  (1993). 
The  window  structure  is  character-based,  which  allows  it  to  be  used  on 
a  variety  of  systems  and  its  code  to  be  adapted  to  other  system 
environments . 

The  development  of  a  user-friendly  interface  for  NCAMD  was  based  on 
the  philosophy  of  Roth,  Di  Stefano,  and  Chang  (1989).  They  found  that: 
(1)  the  simpler  a  system  is  the  more  it  will  be  used,  (2)  the  quicker  a 
system  can  be  learned  the  more  that  will  be  learned  about  its  applica¬ 
tion,  and  (3)  the  system  will  be  simpler  and  easier  to  learn  if  its  use 
is  intuitive.  Therefore,  the  user  interface  of  NCAMD  has  the  following 
characteristics : 

(1)  To  facilitate  the  processing  of  patients  and  to  minimize  the 
impact  of  using  the  system  on  the  IDC's  time,  the  entry  of  data  and 
responses  were  optimized  for  speed. 

(2)  To  allow  for  system  use  by  a  wide  range  of  computer -experienced 
users,  the  system  was  designed  to  function  by  memorizing  a  few  special 
keys  or  by  use  of  a  mouse. 
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(3)  The  system  has  been  simplified  by  making  the  key  and  mouse 
operations  consistent  throughout  the  program.  Key  and  mouse  functions 
are  not  different  at  different  levels  of  the  system. 

(4)  The  system  contains  context-sensitive  flexibility.  Numerous 
correct  and  incorrect  user  responses  are  identified  by  the  system. 

(5)  The  orientation  of  the  user  throughout  the  system  is  aided  by 
means  of  context-sensitive  help.  This  allows  the  user  to  receive 
instructions  when  in  doubt  about  how  to  proceed. 

(6)  Finally,  to  enable  the  user  to  feel  secure  when  using  the  system, 
it  was  designed  to  be  almost  impossible  to  "crash. "  Through  the  use  of 
the  above  characteristics,  it  is  difficult  to  become  lost  or  disoriented 
at  any  level  of  interaction. 

Instrument 


A  self-administered  questionnaire,  the  NCAMD  Rating  Form  (NRF) , 
consisting  of  a  mixture  of  27  open-ended  and  closed-ended  items,  was 
developed  for  use  in  this  study.  The  development  of  the  NRF  was  based 
upon  the  objectives  of  the  study,  available  literature,  and  expert 
opinion.  The  NRF  was  designed  to  gather  information  that  would  answer 
questions  in  the  conceptual  areas  of  system  utility  and  software  and 
hardware  performance.  Some  of  the  items  in  these  conceptual  areas  were 
created  to  address  the  problems  associated  with  the  implementation  of 
NCAMD  programs  that  de  Dombal  (1990a)  has  outlined.  Other  items  were 
designed  to  gauge  the  extent  to  which  the  goals  (see  above,  Roth  et  al., 
1989)  of  developing  a  user-friendly  interface  for  NCAMD  were  achieved. 

Subjects 

Twenty-seven  iDCs  and  3  GMOs  were  solicited  as  voluntary  subjects. 
Nineteen  IDCs  and  1  GMO,  serving  aboard  5  surface  ships  and  8  submarines, 
returned  surveys.  Four  of  the  subjects'  data  were  deleted  from  analyses 
because  of  missing  data.  The  IDCs  ranged  in  paygrade  from  E-5  (Second 
Class  Petty  Officer)  to  E-8  (Senior  Chief  Petty  Officer) ,  had  performed 
on  independent  duty  for  more  than  1  year,  their  length  of  service  ranged 
from  8  to  22  years,  and  they  had  completed  basic  corpsman  "A"  school  and 
the  more  specialized  Independent  Duty  Corpsman 's  School.  The  GMO  was  an 
0-3  (Lieutenant ) and  had  served  in  the  Navy  for  approximately  2  years. 
Additionally,  2  GMOs  (0-3s)  and  2  IDCs  (one  E-7  and  one  E-8)  participated 
in  focused  interviews. 

All  subjects  had  some  experience  with  a  computer.  Four  were  their 
department's  computer  software  specialist,  11  had  done  maintenance  on  the 
department's  system  (6  doing  backups  regularly),  12  had  used  software 
packages  other  than  SAMS  and  NCAMD  (6  respondents  had  used  spreadsheets 
or  word-processing  software) ,  and  2  had  not  used  SAMS  or  other  patient 
management  systems. 

Procedure 


The  IDCs  and  GMOs  were  briefed  on  the  objectives  and  requirements  of 
the  study  and  given  a  copy  of  the  study  protocol,  medical  encounter 
forms,  evacuation  forms,  and  the  CRF.  IDCs  and  GMOs  attached  to  vessels 
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in  which  the  system  could  not  be  installed  by  NHRC  personnel  were  given 
intensive  training  and  familiarization  in  the  installation  and  use  of  the 
NCAMD  system.  IDCs  and  GMOs  attached  to  vessels  wherein  NHRC  personnel 
could  install  NCAMD  were  given  abbreviated  training  in  its  use. 

Results 

Descriptive  results  of  the  NRF  will  be  given  first,  followed  by  the 
results  of  the  focused  interviews.  The  GMO's  responses  were  similar  to 
those  of  the  majority  of  IDCs.  Therefore,  the  GMO's  responses  are 
grouped  with  those  of  the  IDCs,  and  both  are  referred  to  as  respondents 
or  clinicians.  The  results  of  the  NCAMD  evaluation  will  be  separated 
into  the  three  conceptual  categories  listed  in  the  Method  section  of 
system  utility,  software  performance,  and  hardware  performance. 

Modal  and  mean  responses  will  be  given  where  appropriate.  Means  were 
computed  by  coding  the  responses  from  lowest  (1  =  "poor,"  "rarely,"  "not 
accurate  at  all,"  and  "not  useful  at  all")  to  highest  (4  =  "excellent," 
"always,"  "very  accurate,"  and  "very  useful")  of  a  four -category,  forced- 
choice  scale. 

System  Utility 

Five  of  the  items  used  to  assess  the  utility  of  the  NCAMD  system  are 
summarized  in  Table  1.  For  these  items  respondents  were  asked  to  rate 
on  a  four-category,  forced-choice  scale:  (1)  how  often  the  system  was 
used  from  "daily"  to  "less  than  once  a  month,"  (2)  the  usefulness  of  the 
system's  training  mode  from  "poor"  to  "excellent,"  (3)  the  difficulty  of 
generating  a  diagnosis  with  NCAMD  from  "very  easy"  to  "very  difficult," 

(4)  the  accuracy  of  the  system  from  "very  accurate"  to  "very  inaccurate," 

(5)  the  usefulness  of  the  reports  generated  by  NCAMD  from  "very  useful" 
to  "not  useful  at  all."  The  grand  mean  rating  for  these  items  assessing 
the  utility  of  the  NCAMD  is  2.6  of  a  possible  4.0. 

Table  1 

Mean  and  Modal  Responses  Related  to  the  Utility  of  NCAMD  System  (n  =  16) 


Item 

Response 

Mean 

Mode,  n 

Frequency 
of  Use 

"less  than  once  a  month"  (1.2) 

"less  than  once  a  month,"  13 

Utility  of 
Training  Mode 

"good"  (2.8) 

"fair, "  7 

Generating  a 
Diagnosis 

"somewhat  easy"  (2.6) 

"somewhat  easy, "  8 

Accuracy  of 
Information 

"somewhat  accurate"  (3.2) 

"somewhat  accurate, "  9 

Utility  of 
Reports 

"somewhat  useful"  (3.0) 

"somewhat  useful, "  7 

5 


Clinicians  were  asked  to  estimate  the  most-used  to  least-used  NCAMD 
functions.  Nine  said  they  mostly  used  it  as  a  "training  tool,"  4 
said  they  used  it  as  a  "treatment /diagnostic  reference."  Nine  clinicians 
said  that  "Patient  registration"  was  the  least-used  function. 

The  effect  the  use  of  NCAMD  had  on  health  care  was  assessed  along  a 
forced-choice,  three-category  scale  for  five  factors  and  the  results  are 
summarized  in  Table  2.  One  clinician  did  not  respond  to  factors  2  and 
3 .  The  first  factor  was  the  effect  the  system  had  on  the  amount  of  time 
for  a  patient  examination.  The  second  factor  was  the  effect  the  system 
had  on  the  thoroughness  of  the  examination.  Factor  3  dealt  with  the 
effect  the  system  documentation  had  on  health  care.  The  fourth  factor 
concerned  the  ability  of  the  clinician  to  reach  a  diagnosis.  Factor  5 
concerned  the  effect  of  treatment  information  on  health  care. 

Table  2 

The  Effect  of  NCAMD  on  Health  Care 


Factor 

Rating 

Positive, 

n 

Neutral,  n 

Neaative,  n 

1. 

Time  required  for 
examination 

Less, 

0 

0 

Lengthened, 

16 

2  . 

Thoroughness  of 
examination 

More, 

8 

6 

less. 

1 

3  . 

Effect  of  documentation 
on  health  care 

Improved, 

9 

5 

worse, 

1 

4. 

Ability  to  make 
diagnosis 

Improved, 

9 

6 

less , 

1 

5. 

Affect  of  treatment 
information  on  health  care 

Imp roved. 

11 

5 

worse , 

0 

Total 

37(47%)  22(28%) 

19  (24 

The  clinicians  were  asked  to  list  other  diagnostic  modules  they  would 
like  added  to  NCAMD.  Nine  said  they  would  like  to  have  a  dental  module, 
10  a  dermatological  module,  5  a  musculoskeletal  module,  seven  a 
gynecological  module,  5  a  psychiatric  module,  and  9  a  trauma  module. 
Some  respondents  indicated  they  would  like  to  have  more  than  one  of  the 
listed  modules. 

Software  Performance 


Eight  items  were  used  to  collect  information  concerning  the 
performance  of  the  software  in  the  NCAMD  system,  and  the  responses  are 
summarized  in  Table  3.  The  clinicians  were  asked  to  rate  on  a  four- 
category,  forced-choice  scale:  (1)  the  ease  of  learning  the  NCAMD  system 
from  "very  easy"  to  "very  difficult,"  (2)  the  Help  feature  of  the  system 
from  "poor"  to  "excellent,"  (3)  how  often  the  system  prompted  for  correct 
information  from  "always"  to  "rarely,"  (4)  how  often  sign  and  symptoms 
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were  displayed  in  a  meaningful  order  from  "always"  to  "rarely,"  (5)  the 
ease  of  generating  a  diagnosis  from  "excellent"  to  "poor,"  (6)  the 
accuracy  of  the  system  diagnosis  from  "very  accurate"  to  "very  inaccu¬ 
rate,  "  (7)  the  accuracy  of  treatment  information  from  "very  accurate"  to 
"very  inaccurate, "  and  (8)  how  well  the  system  interfaces  with  SAMS  from 
"excellent"  to  "poor."  Overall,  the  grand  mean  rating  for  these  items 
assessing  software  performance  was  2.9  of  a  possible  4.0. 

Table  3 

Mean  and  Modal  Responses  Related  to  NCAMD  Software  Performance  (n  =  16) 


Item  Response 


Mean  Mode,  ri 

somewhat  -easy, "  9 
Fair , "  6 ;  "good, "  6 
usually,  "  9 


Ease  of  Learning 

Help  Feature 

Accuracy  of  Prompt 

Meaningful  Signs 
and  Symptoms 

Ease  of  Diagnosis 

Accuracy  of  Diagnosis 

Accuracy  of  Treatment 
Information 

System  Interface 

Hardware  Performance 


somewhat  easy"  (3.2) 
good"  (2.7) 
usually"  (2.8) 

usually"  (2,9) 
good"  (2.6) 

somewhat  accurate"  (3.1) 

somewhat  accurate"  (3.2) 
good"  (2.5) 


usually, "  10 
good,  "  8 

somewhat  accurate, "  8 

somewhat  accurate, "  13 
good,  "  6 


Information  related  to  hardware  performance  was  collected  with  6 
items  and  focused  interviews.  One  item  asked  respondents  to  indicate 
their  opinion  of  their  computer's  response  time  on  a  four-category, 
forced-choice  scale  from  "poor"  to  "excellent."  The  mean  (M  =  2.2)  and 
modal  response  was  "fair."  The  other  5  items  consisted  of  open-ended 
questions.  Summaries  will  not  be  given  for  these  items  because  many 
clinicians  did  not  respond  and  because  the  purpose  of  the  items  was  to 
solicit  qualitative  criticisms  of  the  system.  Overall,  the  questions 
elicited  a  balance  of  both  positive  and  negative  remarks  concerning  the 
system  and  also  comments  regarding  suggested  improvements  to  NCAMD. 
Regardless  of  future  hardware  additions  to  the  NCAMD  system,  respondents 
said  they  would  like  to  have  a  modem  or  satellite  link  with  a  Battle- 
Group  or  shore-based  physician  and  computers  with  at  least  386  CPUs.  In 
the  area  of  future  peripheral  material,  additions  to  the  NCAMD  system 
respondents  said  they  would  like  chest  pain  data  sheets  and  more 
reference  materials.  Respondents  thought  the  diagnosis  of  "nonspecific 
abdominal  pain"  was  confusing  and  should  therefore  be  deleted  from  the 
system.  A  few  statements  were  received  indicating  the  prompt  to  enter 
electrocardiogram  (ECG)  and  electrolyte  data  should  be  deleted  because 
most  units  did  not  have  the  equipment  to  gather  these  data. 
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Positive  comments  included:  the  system  is  “...useful  in  organizing 
thoughts  prior  to  speaking  to  Battle  Group  Surgeon, "  “easy  to  use  and 
user-friendly,"  “...liked  Help  screens,"  "liked  using  it  as  a  training 
tool,"  "good  trauma  tool,"  and  “fun  for  learning."  Negative  comments 
included:  using  the  system  is  “ .  .  .  time-cons\aming,  “  the  system  is  "...a 
waste  of  time, "  “treating  patient  while  using  the  system  is  constrain¬ 
ing,  "  and  the  system  was  “ . . .too  slow.  . .  “  and  " . . .needs  to  be  faster. . . ,  “ 
and  "it  prevents  the  corpsman,  or  allows  the  corpsman  to  rely  on  a 
program  rather  than  using  the  art  of  medicine  to  diagnose  and  treat  a 
patient . " 

Focused  Interviews 


Focused  interviews  (Merton,  Fiske,  &  Kendall,  1956)  were  conducted 
with  two  GMOs  {0-3s)  and  two  IDCs  (one  E-7  and  one  E-8) .  The  results  of 
the  interviews  were  similar  to  those  of  the  questionnaire  survey; 
therefore,  only  the  results  that  differed  from  those  of  the  questionnaire 
will  be  presented.  The  comments  of  the  interviewees  will  be  summarized 
into  two  categories:  suggested  changes  to  NCAMD  and  general  remarks. 
Because  the  comments  of  the  GMOs  did  not  substantially  differ  from  those 
of  the  IDCs,  they  will  be  presented  as  a  group. 

The  interviewees  suggested  that  printouts  of  disease  and  treatment 
references  and  the  ability  to  write  encounters  to  the  sick-call  log  would 
increase  the  administrative  value  of  the  NCAMD  system.  A  suggestion  was 
also  received  to  add  CD-ROM  capability  to  the  system  to  allow  it  to  store 
and  retrieve  illustrative  material. 

Interviewees  stated  that  the  addition  of  the  following  modules  would 
significantly  increase  usage  of  the  system:  genitourinary  diseases, 
infectious  diseases  of  the  head  and  neck,  and  psychiatry. 

General  comments  received  from  the  GMOs  and  IDCs  included:  “One  or 
two  cases  a  month  are  serious  enough  to  use  CAMD; "  "A  printout  of 
diagnostic  results  would  improve  the  credibility  of  the  IDC  to  the  CO; " 
"CAMD  familiarization  should  be  part  of  IDC  school  curriculum,  stress 
limitation  of  system; “  "Reduce  the  amount  of  stimuli  on  the  screen,  too 
distracting  in  stressful  situations;"  and  "Reference  source  may  be  better 
than  a  diagnostic  aid. " 


Discussion 

For  the  purpose  of  discussion  the  results  of  the  evaluation  of  NCAMD 
system  will  be  separated  into  the  three  conceptual  categories  that  are 
listed  in  the  Method  and  Results  sections.  The  three  categories  are 
system  utility,  software  performance,  and  hardware  performance.  The 
system  utility  category  will  include  responses  to  items  and  interviewee 
comments  related  to  the  general  usefulness  of  the  system  and  the  implica¬ 
tions  of  using  the  system.  Results  related  to  software  performance 
include  responses  to  items  and  interviewee  statements  related  to 
suggested  additions  and  deletions.  The  category  of  hardware  performance 
was  included  because  available  hardware  intrinsically  affects  the  use  of 
NCAMD.  Consequently,  NCAMD  software  cannot  be  evaluated  apart  from 
hardware.  Therefore,  item  responses  and  interviewee  remarks  will  be 
integrated  to  provide  a  summary  of  hardware  performance. 
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System  Utility 


Although  most  of  the  clinicians  stated  the  utility  of  the  system  was 
good  to  excellent,  the  results  suggest  NCAMD  is  being  used  less  than 
monthly,  primarily  for  training,  and  usually  as  a  diagnostic  reference 
^  tool  after  an  initial  diagnosis  has  been  completed. 

The  frequency  of  NCAMD  use  appeared  to  be  directly  related  to  the 
-•  incidence  of  abdominal  and  chest  pain  patients,  the  processing  capability 

of  available  computer  hardware,  and  the  ability  of  the  clinicians  to 
integrate  NCAMD  into  examination  procedure.  Respondents  stated  few  chest 
and  abdominal  pain  cases  are  seen  each  month,  and  the  available  computer 
hardware  is  tediously  slow.  They  felt  using  NCAMD  while  examining  a 
patient  was  difficult  due  to  the  processing  speed  of  available  computers 
and  the  need  to  interact  with  the  patient  and  the  computer  simultaneous¬ 
ly.  Therefore,  when  NCAMD  was  used  it  lengthened  the  time  required  to 
examine  a  patient.  Dallos  and  Cunningham  (1990)  found  the  use  of  NCAMD 
and  structured  forms  with  abdominal  pain  patients  added  about  4  minutes 
to  the  work  load  for  each  patient.  However,  other  studies,  including 
Dallos  and  Cunningham  (1990),  Patterson-Brown  et  al.,  (1989),  and  the 
present  study  found  that  using  NCAMD  in  conjunction  with  patient 
examinations  make  such  examinations  more  thorough.  Finally,  Patterson- 
Brown  and  Vipond  (1990)  found  the  use  of  structured  data  collection  forms 
greatly  improves  the  diagnostic  accuracy  of  clinicians. 

Overall,  little  use  was  made  of  NCAMD  as  a  reference  for  patient 
data.  For  NCAMD  to  be  used  routinely  as  a  patient  reference,  all  patient 
encounters  would  have  to  be  entered  and  the  present  data  show  this  was 
not  done. 

There  was  agreement  that  generating  and  reaching  a  diagnosis  on  NCAMD 
was  relatively  easy,  and  the  information  NCAMD  provided  was  accurate. 
Diagnostic  information  is  related  to  treatment  data  within  NCAMD,  and 
respondents  stated  the  system  provides  better  treatment  information  than 
they  normally  have  available.  Most  respondents  felt  the  documentation 
NCAMD  provided  was  useful  and  generally  improved  the  quality  of  health¬ 
care  services.  This  supports  the  results  of  an  analysis  of  the  legal  and 
ethical  clinical  use  of  computers  done  by  Miller,  Schaffner,  and  Meisel 
(1985)  who  found  computers  should  only  be  used  if  they  improve  the 
quality  of  medical  care. 

The  present  results  suggest  the  utility  and  use  of  NCAMD  would  be 
increased  if  more  reference  materials  were  added  to  NCAMD.  Suggested 
areas  were  dental,  dermatological,  musculoskeletal,  gynecological, 
psychiatric,  trauma,  infectious  diseases  of  the  head  and  neck,  and 
genitourinary.  Nice  (1984)  found  the  leading  cause  for  at-sea  communica¬ 
tions  were  diagnoses  associated  with  traiama,  dental,  abdominal  and  chest 
pain,  infectious  diseases,  mental  disorders,  genitourinary,  and  nervous 
system  and  sense  organs,  respectively.  IDCs  and  GMOs  initiated 

communications  to  elicit  a  consultation  concerning  the  suspected 
diagnoses.  Taken  together,  the  present  results  and  those  of  Nice  (1984) 
suggest  NCAMD  consultations  in  the  diagnostic  areas  of  traima,  dental, 
infectious  disease,  mental  disorders,  and  genitourinary  would  be  of 
particular  value  to  at-sea  clinicians. 
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In  summary,  NCAMD  was  used  more  often  for  training  than  for  patient 
evaluation  or  treatment.  The  two  most  prominent  reasons  for  the  lack  of 
NCAMD  use  in  direct  patient  care  are  the  slowness  of  available  computer 
CPUs  and  the  unavailability  of  pertinent  diagnostic  and  treatment 
information.  NCAMD  utility  and  use  would  be  increased  by  providing 
clinicians  with  computers  with  faster  CPUs  and  easy  accessibility  to  more 
diagnostic  and  treatment  reference  materials.  However,  the  present  value 
of  NCAMD  as  a  training  tool  for  diseases  of  the  chest  and  abdomen  should 
not  be  underestimated.  Many  studies  have  shown  the  use  of  a  CAMD  system 
improves  clinician  performance.  Clinician  performance  appears  to  improve 
due  to  the  use  of  a  consistent  and  structured  method  of  data  collection 
via  a  CAMD  system  that  provides  feedback  and  a  focus  on  education  (de 
Dombal,  1990a,  1990b,  1991) .  Therefore,  the  value  of  the  NCAMD  system 
as  a  diagnostic  tool  is  often  of  indirect  value.  Through  the  use  of 
NCAMD  as  a  training  tool,  the  clinician's  diagnostic  skills  are  improved, 
and  he  or  she  is  then  better  able  to  diagnose  trauma  and  disease. 

Software  Performance 


All  respondents  stated  the  NCAMD  system  was  easy  to  learn.  The  ease 
of  learning  was  probably  related  to  the  user-friendliness  of  the  system, 
which  was  rated  as  “good"  by  the  majority  of  respondents.  Among  the 
prominent  features  added  to  NCAMD  to  aid  in  learning  the  system  were  the 
Help  windows,  which  most  of  the  respondents  stated  were  “good"  to 
"excellent."  De  Dombal  (1990a)  stated  there  are  three  major  reasons  why 
clinicians  do  not  use  NCAMD  systems  more  often.  The  first  reason  is  they 
take  time  to  use,  and  the  second  reason  is  they  require  computer  skills. 
The  third  reason  is  related  to  the  first  two,  and  it  is  the  inability  to 
type  proficiently.  He  further  stated  the  one  area  of  clinical  computing 
that  requires  urgent  action  is  the  development  of  user-friendly 
interfaces.  The  NCAMD  system  has  responded  to  some  of  these  findings 
since  it  requires  minimal  time  to  operate  and,  although  data  entry 
requires  the  use  of  the  keyboard,  most  functions  can  be  accomplished  via 
a  mouse.  Also,  as  proficiency  is  achieved  on  the  system  and  baseline 
data  have  been  entered  for  personnel,  the  system  should  save  time  and 
effort . 

The  perceived  accuracy  of  NCAMD  was  judged  to  be  reliable  and  the 
majority  of  the  respondents  indicated  it  usually  prompted  for  correct 
information.  Generally,  respondents  stated  the  diagnoses  NCAMD  generated 
were  correct  and  treatment  information  was  rated  as  "very  accurate." 
Most  respondents  felt  signs  and  symptoms  were  displayed  in  a  meaningful 
order,  and  this  factor  probably  contributed  to  respondents  finding  it  was 
easy  to  generate  a  diagnosis. 

To  ultimately  use  NCAMD  as  an  administrative  aid  it  has  been 
interfaced  with  SAMS.  Most  respondents  rated  this  interface  as  “good" 
and  some  though  it  was  "excellent."  Respondents  suggested  printouts  of 
disease  and  treatment  references  and  the  ability  to  write  encounters  to 
the  sick-call  log  would  increase  the  administrative  value  of  the  NCAMD 
system. 

The  results  of  the  critique  of  NCAMD  software  suggest  the  software 
is  easy  to  learn,  easy  to  use,  and  the  information  prompted  for  and 
generated  by  NCAMD  is  reliable  and  accurate.  The  diagnosis  of  "non¬ 
specific  abdominal  pain"  was  considered  confusing  and  should  be  deleted 
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from  the  NCAMD.  The  need  to  enter  ECG  and  electrolyte  data  should  be 
deleted  because  small  ships  and  submarines  do  not  have  the  equipment  to 
gather  these  data.  At  present,  if  ECG  and  electrolyte  data  are  unknown 
the  user  is  advised  to  enter  normative  data. 

Hardware  Performance 


Although  NCAMD  users  stated  that  the  overall  computer  response  time 
was  slow,  the  response  time  for  generating  a  diagnosis  was  found  to  be 
fair  to  good.  The  slowness  of  available  computer  systems  was  compounded 
by  the  need  to  manipulate  the  system  and  enter  data  via  the  keyboard 
because  many  sites  did  not  have  a  mouse.  The  reporting  function  was 
minimally  useful,  since  many  of  the  sites  did  not  have  printers. 

Regarding  potential  hardware  additions  to  the  NCAMD  system,  respon¬ 
dents  said  they  would  like  to  have  a  modem  or  satellite  link  with  a 
Battle-Group  or  shore-based  physician  and  computers  with  at  least  386 
CPUs.  Such  a  link  to  a  physician  would  allow  the  sea-based  clinician  to 
make  more  timely  and  informed  decisions  regarding  patient  care. 
Previously,  Nice  (1984)  found  at-sea  clinical  decision-making  would  be 
improved  by  providing  clinicians  with  medical  telecommunications 
technologies  that  transmit  x-ray  images  and  TV  images  of  body  parts .  In 
the  area  of  future  peripheral  material  additions  to  the  NCAMD  system, 
respondents  said  they  would  like  chest  pain  data  sheets  and  more 
reference  materials.  Data  sheets  allow  for  a  methodical  and  thorough 
evaluation  of  patients  while  reference  materials  provide  in-depth  answers 
concerning  symptomatology  and  treatment.  Wellwood  et  al.  (1992)  found 
when  clinicians  recorded  data  on  structured  information  sheets  prior  to 
consultation  with  senior  clinicians  there  was  a  significant  reduction  in 
the  number  of  hospital  admissions  and  fewer  operations  with  negative 
findings.  This  would  suggest  that  if  at-sea  clinicians  were  provided 
with  structured  forms,  a  variety  of  NCAMD  modules,  and  a  link  to  senior 
shore-based  clinicians,  there  may  be  fewer  medical  evacuations  and  fewer 
unnecessary  procedures  may  be  performed  at  sea.  A  suggestion  was  also 
received  to  add  CD-ROM  capability  to  the  system  to  allow  it  to  store  and 
retrieve  illustrative  material.  In  summary,  it  is  evident  sea-based 
clinicians  believe  their  use  of  NCAMD  would  improve  their  performance  if 
it  were  installed  on  updated  computer  hardware.  Therefore,  when  faster 
CPUs  are  available  NCAMD  software  may  gain  greater  acceptance. 

In  response  to  de  Dombal's  (1990a)  proposal  that  seven  problems  must 
be  addressed  before  CAMD  systems  are  successfully  implemented,  the 
present  results  show:  (1)  clinicians  believe  CAMD  systems  enhance  their 
performance;  (2)  NCAMD  uses  a  structured  method  to  gather  patient  data 
and  case  histories;  (3)  NCAMD  is  user-friendly;  (4)  no  preference  for 
deterministic  or  probabilistic  systems  (NCAMD  is  probabilistic) ;  (5) 
NCAMD  gives  the  clinician  diagnoses  in  their  most  probable  order  (i.e., 
the  most  probable  first,  without  giving  numerical  values),  thus  enabling 
the  clinician  to  investigate  diagnostic  options  without  being  biased  by 
CAMD  estimates;  (6)  NCAMD  was  of  value;  however,  educating  clinicians  on 
the  benefits  of  using  CAMD  will  be  a  continuing  problem  that  may  be 
resolved  by  introducing  NCAMD  to  medical,  nursing,  and  IDC  students 
during  their  earliest  training;  and  (7)  IDCs  and  GMOs  are  computer- 
literate  and  adept  at  using  information  technology. 
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Bankowitz  et  al.  (1989)  have  stated  that  evaluations  of  CAMD  systems 
should  focus  on  effectiveness  as  well  as  accuracy.  Previous  studies  of 
NCAMD  have  established  its  objective  accuracy,  and  this  report  has  gauged 
its  perceived  accuracy.  The  goal  of  future  evaluations  will  be  an 
objective  assessment  of  the  impact  of  the  system  on  quality  of  care  and 
patient  outcome  measures.  The  present  results  suggest  that  NCAMD  will 
become  widely  used  when  it  shows  beneficial  reductions  in  length  and  cost 
of  hospitalization,  in  the  time  required  to  establish  an  accurate 
diagnosis,  and  in  the  use  of  health-care  resources  during  the  diagnostic 
process . 
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